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Critical Examination of the Use of Online Technologies in Diverse Courses at 

a Large Comprehensive University 



James M. Monaghan 
Rowena S. Santiago 
California State University 



Abstract 

We explore cases of three classes that implemented online teaching / learning technologies as part of a university wide 
faculty development grant program. We examined students* satisfaction with key components of the learning experience. 
Instruments included students’ pre-assessments and post assessments, an instructors’ post survey, course syllabi, grant 
proposals, and instructors’ end of grant reports. We present implications for campus wide implementation of online teaching/ 
learning technologies. We include discussion of a model for implementing innovative technologies campus wide (online teaching 
support model). 

Background 

In 1997, faculty interest in online teaching at California State University San Bernardino (CSUSB) started increasing. By 
1999, the number of faculty who wanted to do online teaching had grown significantly. Faculty faced many questions and issues 
as they embarked to do online teaching. The Teaching Resource Center (TRC), the faculty development unit that support 
teaching and innovative instruction, also faced the task of helping in faculty’s new role as online teachers. 

To address these needs, TRC drew upon instructional design and evaluation models (Van Slyke, Kittner & Belanger, 1998; 
Belanger & Jordan, 2000; Dick & Carey, 1996; Salisbury, 1996; Seels & Glasgow, 1998) and identified the major steps involved 
in course development as applied to online teaching. TRC developed a systematic and holistic plan that serves both as the map 
and the glue that holds the various phases and players together (Santiago, 2001). This systematic plan has 7 major steps: 




Figure 1. Online Teaching Support Model: 7 Major Steps 

Planning and Preparation. The instructor studies the big picture and the instructional and non-instructional issues that will have 
an impact on successful online teaching. Addressing issues up front gives a good estimate of how much time and effort need to 
be invested, and helps determine one’s readiness for online teaching. 

Funding . Faculty are encouraged to seek funding through grant programs that lead to a course buyout, money for resources 
and/or hire a student assistant 

Instructional Design and Training: Designing lessons for online teaching is not simply converting existing materials into digitized 
format. Skills and training for goal analysis, task analysis, assessment, instructional materials development, and the use of 
hardware and software are addressed in this step. 

Instructional Materials Development and Testing . This step involves developing instructional materials, testing them, and 
making sure that they can be delivered successfully online, and that users will be able to access them with minimal, if not totally 
free of instructional and technological glitches. 

Implementation. The course is actually taught online. The instructional design is put into action, the instructional materials are 
used, and technology is relied upon to deliver instruction, all towards the achievement of learning goals. 
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Evaluation . Evaluation identifies the strengths and weaknesses of the instructional design, of instructional materials, and delivery 
medium. Most importantly, it measures the extent to which goals were met. 

Dissemination . Innovation and change that is well planned and appropriately evaluated also results in lessons learned and quality 
products. When reported to a professional community, this leads to further validation, peer evaluation, and replication, which 
then leads to scholarship of teaching (Hutchings and Schulman, 1999). 



Description of Grant Program 

To support online teaching, TRC sponsors the Web-based Course Development Grant program that awards summer stipend 
($5,000) to faculty. The course redesign or development could range from using a significant combination of online features as 
teaching enhancement, or to deliver the whole course online. When first offered in 1999, 12 faculty proposals were funded. The 
Course Development Grant Program was funded by the Office of the Provost. Administrative support for online teachingalso 
resulted in student support through the Student Technology Support Center, the establishment of an Office for Distributed 
Learning and the hiring of a librarian to support distance courses. 



Analysis of the Efficacy of the Grant Program 

To inform our analysis of the efficacy of the online components of the courses, we utilized a variety of assessment tools 
including pre and post surveys of students, a post survey of faculty, and an analysis of course syllabi, grant proposals and final 
reports. 

Comparison data from the three classes in which we had sufficient data from both the student pretest and posttest are listed 
below. We report results from a comparison of the uses of online technologies in the three courses. We examined student data 
(pretest and posttest), faculty data (free response and multiple choice questions), course syllabi, and the course development 
proposal. Our intention was to utilize triangulation in an attempt to ascertain factors that led to successful and efficacious uses of 
online technologies from both the faculty and the student perspectives. 

Three classes that were offered in the 1999-2000 academic year were examined. The description of each course is listed as it 
appeared in the course syllabi and/or the catalog of programs: 

1 . English as a Second Language (EESL) - Research in English as a Second Language: Covers various perspectives used to 
guide research in TESOL. Includes analysis, discussion, reflection and writing about key issues and concepts in research. 
Students will design and carry out literature review working towards a theoretical framework to gain foundation as expert 
practitioner or future researcher. 

2. Criminal Justice (CJUS) - Research Methods in Criminal Justice: Introduction to scientific methodology and research 
designs used to conduct basic and applied research in the criminal justice field. Emphasis on scientific operationalization, 
survey methodology, and concepts of evaluation design. 

3. Information Sciences (INFO) - Decision Support Systems: Formal hformation systems that support organizational 

decision-making. Topics include the strategy, framework, design, implementation and evaluation of decision support 
systems. Students will create and apply decision support systems to planning, coordinating, organizing, controlling and/or 
directing tasks. 



Pre-Survey 

In the student pre survey, we obtained data on three categories: student demographics, student background, and student 
experience on the use of technology. A summary of demographic data for 61 students who took the pretest in the 3 classes is 
listed below. 

Table L Student Demographics 



Demographics 



Level 



Degree sought 



Age 





CJUS 


EESL 


INFO 


Graduate 


3% 


91% 


5% 


Senior 


40% 




90% 


Junior 


47% 




5% 


Masters 


10% 


100% 


15% 


Bachelors 


90% 




75% 


40’s & 50’s 




27% 


20% 


30's 


7% 


45% 


15% 


20's 


87% 


27% 


65% 



Gender 


Male 


43% 


18% 


45% 




Female 


53% 


82% 


55% 


Miles from campus 


> 50 mi. 


10% 


18% 


5% 




40-49 mi. 


3% 


9% 


10% 




30-39 mi. 


17% 


0% 


20% 




20-29 mi. 


17% 


9% 


5% 




10-19 mi. 


20% 


0% 


15% 




< 9 mi. 


27% 


55% 


40% 



Table 2. Student Background 
Background 

Reason for taking course 



Hours to spend on course 



Same instructor before 





CJSU 


EESL 


INFO 


Recommended 


27% 


9% 


25% 


Required 


83% 


100% 


70% 


Interesting 


17% 


18% 


30% 


Fits schedule 


37% 


18% 


60% 


>= 17 hrs 


3% 


36% 


0% 


13-16 hrs 


3% 


9% 


0% 


9-12 hrs 


23% 


36% 


10% 


5-8 hrs 


57% 


18% 


75% 


0-4 hrs 


13% 


0% 


15% 


Yes 


37% 


73% 


10% 
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As shown above, the majority of the students were in upper division working for a bachelor’s degree except for the Education 
masters students (EESL). Most of the students were in their 20’s and 30’s. Female students were in the majority, especially in 
the Education course. Almost half of the students live close to campus (< 9 miles) except for the CJUS students who are 
distributed in various distances from campus. 

For background data, we surveyed students’ reasons for taking the course, the number of hours that they intended to spend on 
the course per week, and whether they have had the instructor before. Data from this component of the survey is listed below. 

In general, students were taking the course because it was required. A secondary reason was that it fits their schedule. The 
majority of the undergraduate students planned to spend 5-12 hours per week on the course, with graduate students planning to 
spend more than 17 hours per week on their course. Most of these graduate students have had a course with the instructor before. 

We also surveyed students’ use of and comfort level with online technologies. A summary of students’ use of technology is 
given below: 

Table 3. Student Use of Technology 



Use of technology 
Access at home 



Used in previous course 



Comfort level 
(5=highest) 



Computer 

Internet 

Online syllabus 
Online Lectures 
Online Tests 
Online Research 
Online Discussion 
Email 

Chat Rooms 

Using computers 
Using internet 
Using Chat Rooms 
Using Email 
Taking Online Course 
Joining Online Discussion 



CJUS 


EESL 


INFO 


93% 


91% 


100% 


87% 


91% 


90% 


47% 


73% 


40% 


33% 


73% 


30% 


7% 


82% 


20% 


70% 


91% 


80% 


10% 


36% 


20% 


77% 


91% 


80% 


23% 


9% 


25% 


4.3 


4.0 


4.8 


4.3 


3.9 


4.7 


3.2 


3.4 


4.1 


4.1 


4.6 


4.6 


3.2 


3.8 


3.6 


2.8 


3.5 


3.7 



Based on the data, student access to computers and Internet at home is very high (87-100%). Of the three groups, the 
graduate students had the most experience in the use of online technology in a course. Online experience is high for research and 
email use in all three courses. Students are most comfortable in the use of computers, Internet and email. 



Post-Survey 

In the student post survey we coupled the internally developed and tested Student Evaluation of Learning Effectiveness (or 
SELE, a self report measure of learning that utilized a 5 point Likert scale and which had been approved for use in evaluation of 
teaching by the Faculty Senate), with a self-report instrument concerning technology use in the course. 

One particularly striking result involves the post survey item concerning how well the online learning experiences fostered 
interaction and teamwork among class members. In INFO, this was unmistakably the most highly rated item (mean = 4.5 of 5). 
In the other two courses, EESL and CJUS, this was clearly the lowest rated item (mean = 3. 1 of 5, mean = 2.9 of 5, respectively). 
In comparing scores of both EESL and CJUS with INFO, two tailed t-tests indicated significance with p<. 01. 
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Table 4. Online Components of Course Fostered Interaction and Teamwork 



Course fostered interaction and teamwork 




CJUS 


EESL 


INFO 




5=Excellent 


29% 


18% 


59% 




4=Very Good 


0% 


9% 


35% 




3=Satisfactory 


21% 


45% 


6% 




2=Poor 


29% 


18% 


0% 




1=N/A 


21% 


9% 


0% 




Mean 


2.9 


3.1 


4.5 


We found the highest mean posttest scores in the answers to three questions that related to the structure of the courses 


involvement of the teacher in the learning process. 

Table 5, Satisfaction With Course Items Controlled by Instructor 

Matched objectives with assessment 5 = Excellent 


CJUS 

50% 


EESL 

27% 


INFO 

41% 




4 = Very Good 


36% 


36% 


47% 




3 = Satisfactory 


7% 


36% 


12% 




2 = Poor 


0% 


0% 


0% 




1 =N/A 


7% 


0% 


0% 




Mean 


4.2 


3.9 


4.3 


Utilized clear grading criteria 


5 = Excellent 


43% 


55% 


47% 




4 = Very Good 


36% 


0% 


41% 




3 = Satisfactory 


14% 


27% 


12% 




2 = Poor 


0% 


18% 


0% 




1 =N/A 


7% 


0% 


0% 




Mean 


4.1 


3.9 


4.4 



Instructor contributed to learning 


5 = Excellent 


43% 


36% 


35% 




4 = Very Good 


50% 


36% 


53% 




3 = Satisfactory 


0% 


27% 


12% 




2 = Poor 


0% 


0% 


0% 




1 =N/A 


7% 


0% 


0% 




Mean 


4.2 


4.1 


4.2 
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Table 6, Did Online Components Foster Discussion on Multicultural and Diversity Issues? 
Fostered discussion on multicultural and diversity issues 





CJUS 


EESL 


INFO 


5 = Excellent 


29% 


27% 


18% 


4 = Very Good 


14% 


18% 


35% 


3 = Satisfactory 


21% 


36% 


29% 


2 = Poor 


14% 


18% 


18% 


1 =N/A 


21% 


0% 


0% 



To the question: “The course matched objectives with assessment”, the mean for CJUS was 4.2, for EESL, it was 3.9, and for 
INFO, it was 4.3. “Utilizing clear grading criteria was also rated high by all three courses, with INFO mean at 4.4, CJUS at 4.1 
and EESL at 3.9. All three courses had almost the same means for the question “Instructor contributed to learning” (CJUS = 4.2, 
EESL = 4.1, INFO = 4.2). In tandem, this data would be an indicator of clear grading criteria that connected with stated 
objectives of the courses. 

The lowest means reflected responses to whether the online components of the course “Fostered discussion on multicultural 
and diversity issues”. 

When it came to identifying who contributed to the learning experience, the instructor was ranked highest, based on the mean 
scores. EESL ranked “self’ qually with the instructor, while CJUS ranked “self’ as second and other students as lowest. 
Interestingly, concerning who contributed to their learning, INFO ranked “Other student/s” equally with the instructor. For both 
CJUS and EESL this was the lowest rated item of the three. 

Table 7. Rating For How Well the Instructor, the Student, and Other Students Contributed to the Learning Experience 



Contributed to learning experience 


CJUS 


EESL 


INFO 


Instructor 


4,2 


4.1 


4.2 


Self 


3.6 


4.0 


4.0 


Other Student/s 


3.4 


3,5 


4.2 



We also surveyed how satisfied students were with online course materials. Satisfaction was rated high by all three courses. 
However, when asked if on line materials helped facilitate learning, means for all three courses were low. 

Table 8. Online materials survey results 





CJUS 


EESL 


INFO 


Satisfied with online materials 


4.7 


4,1 


4.2 


Online materials facilitated learning 


2.3 


3.4 


3.5 



For all 3 courses examined, online materials were reported to have saved students time. However, in the EESL class, 
students reported spending more time on the online tests but saving time through use of discussion board. 
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Table 9: Time Spent/Saved Using Online Components 



Time Spent 



Online Materials 



Online Tests 



Discussion Board 





CJUS 


EESL 


INFO 


Saved time 


36% 


64% 


69% 


Spent more time 


0% 


9% 


6% 


No answer 


64% 


27% 


13% 


Saved time 


0% 


27% 


0% 


Spent more time 


0% 


55% 


0% 


No answer 


100% 


27% 


100% 


Saved time 


7% 


64% 


0% 


Spent more time 


0% 


9% 


0% 


No answer 


93% 


27% 


100% 


, Grant Reports, 


, and Post 


Surveys 





The faculty post survey was composed of items concerning their needs and uses of online technologies, coupled with an 
evaluation of students' learning that paralleled the student post survey. We also examined how the use of online technologies 
impacted faculty workload. 

In our ongoing analysis we are considering the factors that led to successful and efficacious use of online technologies for 
both faculty and students. We are currently conducting our analysis via triangulation where we compare student data, faculty 
data (composed of free response and multiple choice questions), information provided in the course syllabi, the grant proposals, 
and the end of grant reports. 

We were not able to directly correlate the faculty post data with the student post data for individual courses due to the 
anonymity of the faculty post surveys. However, we were able to glean some data in the aggregate for professors’ general 
satisfaction with the use of online technologies in their classes. Based on that data it was clear that faculty appreciated the 
support that was given to them. 

Two of the three instructors provided access to syllabus information. Based on this information, in the INFO class, students 
were required to work in teams to complete projects. The professor used online materials but did not list use of the discussion 
board in his syllabus. By contrast, in the EESL class, students were not required to work in teams. The instructor required the 
use of the discussion board, online materials and online tests. 

Table 10: Online components of Course Design 



COURSE 


Based on Proposal: 




Based on syllabus: 


Based on final report: 




DESIGN 




















CJUS 


EESL615 


INFO 


CJUS 


EESL INFO 


CJUS 


EESL 


INFO 






EESL 














Online 


AH course 


Informatio Course 




Assignme 




Course 


Course 


Materials 


notes. 


n 


content. 




nt 




materials 


content 




exercises 


modules/ 


glossary. 




descriptio 












tutorials 


site links 




ns 












using 

PowerPoi 


for articles 
















nt and 
audio 














Online 


Practice 


Part of 


Online 




100-point 




Quizzes 




Tests 


quizzes. 


tutorial 


quizzes 




quiz set. 










assessment 




and 




review 














surveys 




quizzes 








Online 










Accessing 








Research 










TESOL 

research 

via 

technolog 









y 
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Online 

Discussion 

Email 

Chat 

Rooms 



Bulletin 

board 



Mentioned 




Mentioned 


in the 




it affords 


justificatio 




students to 


n but not 




interact 


part of 




but was 


project 




not 


descriptio 




reported 


n 




to have 






been done 




Weekly 


Used 




announce- 


extensively 




ments 





In the grant proposals, teaching was to be enhanced with the posting of online materials and links to relevant websites (see 
figure 10). EESL also incorporated the use of online tests and discussion board. INFO proposed to implement more technologies 
than were actually incorporated into the course. 



Discussion 

Based on the data presented above, it appears clear that in all three courses, students were satisfied with the connections made 
between the objectives of the course and the assessments used. This would imply that the courses were relatively well designed. 

However, though the courses were relatively well received, it appears that improvements could have been made in the 
effective use of online technologies. For instance, the INFO course did not, according to the syllabus, the final report and the 
students’ post surveys, use a discussion board for collaboration and file sharing. This was the case even though extensive 
collaboration was necessary for successful completion of course projects and even though at least 40% of the students lived 20 or 
more miles from campus. 

Myer (2001) indicated that a fundamental problem with much research on online instruction is that it has failed to 
disaggregate the effects of the instructional design from the effects attributable to the technology. This appears to be the case in 
the courses that we examined above, as each could be shown to have been efficacious (based on student post survey data). 

It is striking that two of the three courses did not reflect good scores on online learning activities fostering interaction and 
teamwork. This is striking as online technologies such as threaded discussion boards and chat rooms can be very powerful tools 
for collaboration (see Jonassen, et al., 1999). This would suggest that the full power of the technologies available through the 
WebCT tools to the instructors had not been utilized. In the third course, the INFO course, it is not clear whether chat rooms 
were utilized by students (see table 10). However, the structure of the course required extensive collaboration on projects 
(according to the course syllabus). 

Similarly, for all three classes, the mean scores on facilitating discussions concerning diversity were low. Online 
technologies have the potential to enable ready access to global perspectives (see Papert, 1998), yet it would appear that this 
characteristic of the technology was not utilized effectively. 

The fact that students in all three classes generally believed that using online technologies saved time would suggest 
improved instructional efficiency for all classes. With a population composed of commuters, who often hold full time jobs, this 
result is weighty. 

Though we asked students to rate how well the online learning experiences (using 5=excellent; 4=very good; 3=satis factory; 
2=poor; l=not applicable), contributed to the effectiveness of the class (as measured by multiple items detailed in the above 
tables of results), respondents may not have read the items with respect to online technologies but rather with respect to the entire 
class. For instance, in the INFO class, students identified online components of the class to have contributed to fostering 
interaction and teamwork, yet a discussion board was not utilized in the course, according to the proposal, the syllabus, the final 
report, and student post surveys. We are also concerned that in some cases, students may not have understood the terms used and 
may have therefore answered questions erroneously (e.g. some students rated discussion boards and chat rooms that did not 
appear to have been implemented in a particular course). 

Given our data analysis, several implications are suggested for implementation of the Online Teaching Support Model at our 
university. In examining the 7 components of the model, there is evidence that several of the components may require substantial 
additional support. Specifically, the instructional design and training area may need additional attention to enable faculty to best 
integrate online technologies into their course design. We are concerned that our evidence suggests that instructors may not use 
the most powerful or appropriate technologies to assist with learning outcomes for their class or that they may use technologies in 
ways that are not effective. Additionally, in the evaluation area, several flaws were present. Faculty did not administer the 
instruments consistently in their courses (even though evaluation was a critical component of the grant). Thus, of the twelve 
funded projects, we were able to realize comparison data from only three classes in which we had sufficient data from both the 
student pretest and posttest. 
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We surmise that, in general, there is a tendency for faculty to focus on innovation and on technology and not on evaluation of 
the online course components. In order to better collect data from faculty, and to encourage them to have students participate 
fully in pretest and posttest administration, additional faculty incentives and reminders are likely to be necessary. Engaging 
faculty in a post -course interview would also likely assist analysis of the efficacy of the online course components. 

We will apply the lessons learned from this analysis to future faculty development grant projects. This will assist us in 
serving the educational needs of our students through the use of technologies that can bridge barriers to access. We believe that 
the Online Teaching Support Model can be effectively applied at other institutions that would internally fund faculty 
development efforts. Based on our experience, special attention should be applied to the instructional design and evaluative 
components of such efforts. 
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